A Many-facet Rasch Model to Detect Halo Effect in Three Types of Raters

Raters play a central role in rater-mediated assessment, and rater variability manifested in various forms including rater errors contributes to construct-irrelevant variance which can adversely affect an examinee’s test score. Halo effect as a subcomponent of rater errors is one of the most pervasive errors which, if not detected, can result in obscuring an examinee’s score and threatening validity and fairness of second language performance assessment. To that end, the present study is an endeavor to detect halo effect in L2 essays, using a relatively newly employed methodology, a many-facet Rasch model (MFRM) in language assessment. The participants in this study consisted of 194 raters—subdivided into self-rater, peer-rater, and teacher rater—who rated 188 essays written by 188 undergraduate Iranian English majors at two state-run universities in Iran. The collected data were rated using a 6-point analytic rating scale and were analyzed using the latest version of Facets 3.68.0 to answer the research question of the study. The results of facets analysis showed that, at group level, the raters did not exhibit any sign of halo effect, but, at individual level, all rater types displayed considerable halo effect. Further analysis revealed that rater types were unanimous about halo effect on four items and that self-rater showed more of a halo effect compared to the other two rater types.